1. Field of the Invention
The present invention relates to a picture decoding/display unit which decodes encoded picture data and outputs decoded pixel data for display on a display unit, and more particularly, it relates to a picture decoding and display unit which decodes predictively encoded moving picture data and outputs the decoded data for display. More specifically, the present invention relates to a structure for reducing the storage capacity of a storage element storing B pictures in decoding along the MPEG (moving picture experts group) standard.
2. Description of the Background Art
In transmission of an enormous quantity of picture data, the data quantity is reduced by decoding the picture data in high efficiency. Techniques of such high efficiency encoding include an inter-frame (or inter-field) predictive encoding system. This inter-frame (or inter-field) predictive encoding, which utilizes temporal correlation between pictures, is generally employed for encoding moving picture data.
The inter-frame (or inter-field) predictive encoding includes the following procedure: A screen picture (hereinafter simply referred to as "picture") which is temporally approximate to a current screen picture (frame or field) to be encoded is employed as a reference picture for predicting values of the pixel data of the current picture. Differences (prediction errors) between the predicted values and the pixel data of the current picture are obtained, and then the prediction errors are encoded. The predicted values are formed by data of reference picture pixels (generally pixels of a motion-compensated reference picture in moving picture encoding) corresponding to the current picture pixels to be encoded. When the pictures have high temporal (time) correlation, the difference (prediction error) is small and hence the quantity of data to be transmitted can be reduced to implement effective compression of information.
Systems of such predictive encoding include the MPEG standard which is directed to moving pictures. Systems of encoding and decoding picture data along the MPEG standard, which are explained in Nikkei Electronics, Mar. 14, 1994, pp. 82 to 116, for example, are now briefly described.
Pictures include I, P and B pictures, and the predictive system as employed depends on the type of the pictures.
FIG. 27 illustrates exemplary temporal (timewise) arrangement of pictures. This figure representatively shows nine pictures G1 to G9. The picture G3 is an I picture, which is subjected to intra-frame or intra-field encoding so that its pixel data themselves are encoded. The pictures G6 and G9 are P pictures, which are subjected to inter-frame or inter-field predictive encoding through past reproduced pictures. The pictures G1, G2, G4, G5, G7 and G8 are B pictures, which are predictively encoded through either past or future reproduced pictures or both of these pictures. These B pictures are predictively encoded through combinations of the I and P pictures or of the P pictures. The B pictures are not employed as reference pictures in predictive encoding and decoding. Referring to FIG. 27, the B pictures G1 and G2 are predictively encoded through the future I picture G3 which is temporally subsequent thereto. The B pictures G4 and G5 are predictively encoded through both of the I picture G3 which is a past reproduced picture and the P picture G6 which is a future reproduced picture temporally subsequent thereto. The B pictures G7 and G8 are predictively encoded through the P pictures G6 and G9 which are past and future reproduced pictures respectively.
FIG. 28 illustrates the structure of a single picture. Referring to FIG. 28, the single picture (field or frame) 1020 is divided into a plurality of segments called macro blocks. For the purpose of simplification, FIG. 28 illustrates such an exemplary structure that the picture 1020 is divided into 32 macro blocks MB#1 to MB#32. Processing of a moving picture is generally executed in units of the segments called macro blocks, in both encoding and decoding. Each of the macro blocks MB#1 to MB#32 generally includes 256 pixels arranged in 16 rows and 16 columns, as shown in FIG. 28. Therefore, the picture 1020 shown in FIG. 28 is formed by 128 by 64 pixels. The MPEG standard defines such conditions that the picture is structured by not more than 720 pixels per line (scanning line), not more than 576 lines per frame, and not more than 30 frames per second. However, DCT (discrete cosine transformation) processing and IDCT (inverse discrete cosine transformation) processing are performed in units of blocks of 8 by 8 pixels.
FIG. 29 schematically illustrates the structure (syntax) of a bit stream (a plural bit width) of picture data along the MPEG standard. Referring to FIG. 29, the bit stream is divided into a plurality of layers including a sequence layer, a GOP (group of pictures) layer, a picture layer, a slice layer, a macro block layer and a block layer in the order from the uppermost layer.
The block layer is formed by a block 1100 including a region 1100a including data of DCT coefficients and a region 1100b storing an end of block (EOB) data indicating the end of the block. The region 1100a stores DCT coefficient data of pixels of 8 rows and 8 columns serving as a unit of the DCT processing. When the last AC coefficient of the region 1100a is a nonzero coefficient, the end of block EOB of the region 1100b may not be employed in the block 1100. The DCT processing is performed to reduce spatial redundancy (high correlationship between adjacent pixels) in the picture thereby reducing the picture data quantity in encoding. DCT coefficients can be made localized in a low frequency coefficient region by the DCT processing. It is possible to enlarge values of low frequency components and substantially zeronize values of high frequency components in both of horizontal and vertical directions in space frequencies by quantizing the DCT coefficients. Thus, the picture data quantity is reduced.
The macro block layer is formed by a block 1110 including a prescribed number of (six) blocks 1100 and a macro block header 1115 storing attributes of the data of the macro blocks, motion vectors and the like.
The slice layer includes a slice 1120 which is formed by one or a plurality of macro blocks which are concatenated in picture scanning order. A slice header 1125 storing information indicating the vertical position of the slice on the screen and information such as a start code having a prescribed pattern indicating starting of this slice is provided at the head of the slice 1120.
The picture layer includes a picture 1130 which is formed by a plurality of slices 1120. A picture header 1135 storing information indicating the type (I, P or B picture) of the picture and a start code indicating starting of the picture is arranged at the head of the picture 1130.
The GOP layer includes a GOP 1140 including a plurality of pictures 1130. The pictures 1130 included in the GOP 1140 include at least one I picture and zero or a plurality of P or B pictures. A GOP header 1145 storing a start code indicating starting of the GOP and information such as a flag indicating that this GOP requires no reference from picture data of a GOP which is precedent thereto is arranged at the head of the GOP 1140.
The sequence layer includes a sequence 1150 which is formed by one or a plurality of GOPs 1140 each including one or a plurality of pictures 1130. A sequence header 1155 storing information such as the format of the screen is arranged at the head of the sequence 1150. This sequence header 1155 can be arranged at the head of every GOP 1140 which is included in the sequence 1150, in order to allow reproduction of pictures from an intermediate portion of the sequence. The sequence header 1155 stores information such as a start code having a prescribed pattern indicating starting of the sequence, horizontal and vertical sizes of the pictures, the picture rate (picture display speed), the bit rate, and the B picture cycle (the number of B pictures which are arranged between I and P pictures or between two P pictures).
FIG. 30 schematically illustrates the structure of a picture decoding unit. Referring to FIG. 30, the picture decoding unit includes a decoding circuit DC which receives encoded picture data (bit stream) from an encoding unit and carries out processing which is inverse to that in encoding for restoring original pixel data, and a memory device MR for storing the picture data restored in the decoding circuit DC. The encoded data which are supplied to the decoding circuit DC are predictively encoded data, as described above. In inter-frame predictive encoding, difference between pixel data of a current frame and a predictive frame (reference frame) is encoded. In order to perform decoding processing in the decoding circuit DC, therefore, it is necessary to carry out an operation which is inverse to that for obtaining the difference, i.e., an adding operation, through the predictive frame (reference frame). The memory device MR is provided for storing the reference frame picture data. A B picture employs two pictures as reference pictures, and hence the memory device MR must have a capacity for storing pixel data of at least two pictures. The memory device MR includes banks #1BK1 and #2BK2 storing two pictures respectively. In the following description, it is assumed that the encoded picture data are in a frame structure formed of a frame including both of pixel data of even and odd fields as a unit. In this case, each of the banks #1BK1 and #2BK2 has a storage capacity for storing picture data of one frame.
As hereinabove described, the picture data are encoded and decoded in units of macro blocks. As shown in FIG. 31A, macro blocks MB of a picture 1030 are successively encoded in the order of numbers shown in the figure. Referring to FIG. 31A, the picture 1030 is divided into N and M macro blocks in the horizontal and vertical directions respectively. For convenience, each group of N macro blocks MB which are aligned along the horizontal direction is called "one macro block line" in the following description. In each macro block line, the encoding processing progresses rightward from the macro block which is positioned on the left end of the screen. After encoding of the final macro block in a certain macro block line, such as a macro block MB(N), for example, is completed, a macro block, such as a macro block MB(N+1), for example, which is positioned on the left end of the next macro block line is encoded. The encoded picture data are transmitted from the encoding unit in the order of the encoding of the macro blocks, as shown in FIG. 31B.
The decoding circuit (see FIG. 30) decodes the encoded data which are supplied in the sequence shown in FIG. 31B, similarly in units of the macro blocks MB. Therefore, an output sequence of the decoded picture data which are outputted from the decoding circuit to the memory device is identical to that shown in FIG. 31B.
On the other hand, a general display unit displays a picture on a unit of "pixel line" from the top to the bottom of the screen in accordance with a "raster scan" system. The term "pixel line" indicates a set of pixels which are aligned along the horizontal direction on the screen.
FIG. 32 illustrates a display sequence of pixel data on the screen in the case of displaying a picture in a non-interlace system. Referring to FIG. 32, a picture 1040 of one screen includes 2L pixel lines. Display order of the pixel lines is indicated on the left side of the screen picture 1040. In the case of non-interlace display, the pixels are successively displayed from that on the left upper end of the screen picture 1040. Namely, the display rightwardly progresses on the screen from the pixel which is arranged on the left end in each pixel line. When display of the final pixel (pixel on the right end of the picture 1040) is completed in one pixel line, the pixel on the left end of the next (adjacent) pixel line is displayed. This display sequence is repeatedly executed until display of the final pixel line (the pixel line denoted as 2L) is completed.
FIG. 33 illustrates a display sequence of a picture in an interlace display system. Also in FIG. 33, a picture 1040 includes 2L pixel lines. In this interlace display system, pixel data of the uppermost pixel line on the picture 1040 are first displayed. Then, the next pixel line is interlaced so that pixel data of the second adjacent pixel line are displayed. Thereafter every other pixel lines are interlaced to display the pixel data of the picture 1040. When the process reaches the lower pixel line L of the picture 1040, it returns to the upper side of the picture 1040, to display the pixel data of the interlaced pixel lines. When display of the pixel data of the interlaced pixel lines is completed, a next frame picture is displayed. Namely, the pixel data of the odd pixel lines of the picture 1040 are first displayed in this interlace display, and then the pixel data of the even pixel lines are displayed after display of the pixel data of all odd pixel lines is completed. In the case of the interlace display, one frame is formed by odd and even fields, so that display of the even field is carried out after completion of display of the odd field.
Comparing FIG. 31 with FIGS. 32 and 33, it is clearly understood that the output sequence of the decoded data is different from the display order of the pixel data which are displayed on the display unit. Therefore, a function of converting the sequential order of the pixel data is required for a unit having picture decoding and display functions. The function of converting the arrangement order of the pixel data is implemented through the memory device. Namely, the pixel data are written in the memory device in the order of the decoding of pixel data, while they are read along the order of the displaying of pixel data. In other words, this conversion function is implemented by decoding supplied encoded picture data, writing the decoded data in the memory device in the order of the macro blocks on a macro block basis, and reading the pixel data from the memory device pixel by pixel along the order of the non-interlace or interlace system in accordance with the display system of the display unit.
FIG. 34 schematically illustrates the structure of a conventional decoding and display unit having functions of decoding and displaying picture data. Referring to FIG. 34, the conventional decoding and display unit includes a decoding/display circuit DDC which decodes pixel data included in supplied encoded picture data (bit stream), restores original pixel data and outputs the restored data to a display unit DP for display, and a memory device MR for storing reference picture data employed in the decoding processing in the decoding/display circuit DDC and pixel data for display. The memory device MR includes a bank #1 (BA1) for storing I or P pictures, a bank #2 (BA2) for storing pixel data of P pictures, and banks #3 (BA3) and #4 (BA4) for storing pixel data of B pictures. A picture data processing sequence of the decoding/display unit shown in FIG. 34 is now described with reference to FIG. 35, which is a timing chart thereof.
The decoding/display circuit DDC is supplied with pixel data of respective pictures in the order which is different from that in display. Pixel data of temporally subsequent pictures, i.e., future pictures, are employed in decoding of B pictures, and hence the future pixel data are transmitted in advance of B pictures. The decoding/display circuit DDC successively decodes the picture data which are supplied from a transmission side in accordance with the transmission order. In display, pictures are transposed into the display order along the types (I, P, B) of the pictures, so that the pixel data of the pictures are read in the order for display and supplied to the display unit DP. It is assumed that the banks #1 to #4 (BA1 to BA4) of the memory device MR store no valid pixel data at present.
In a period T1, the decoding/display circuit DDC is supplied with picture data of an I picture I3, to decode the data. The decoded picture data of the I picture I3 are written in the bank #1 (BA1) of the memory device MR. Upon completion of the period T1, the bank #1 stores all pixel data of the I picture I3.
In a next period T2, the decoding/display circuit DDC is supplied with encoded data of a B picture B1. The decoding/display circuit DDC decodes the encoded data of the B picture B1, and writes the decoded pixel data in the bank #3 (BA3). Upon completion of the period T2, the bank #3 (BA3) stores all pixel data of the B picture B1.
In a period T3, the decoding/display circuit DDC is supplied with encoded data of a next B picture B2. In synchronization with starting of a decoding processing operation for the B picture B2 in the period T3, the pixel data of the B picture B1 are read from the bank #3 (BA3), supplied to the display unit DP and displayed thereon. Namely, decoded picture data of the B picture B2 are written in the bank #4 of the memory device MR while the pixel data of the B picture B1 are read from the bank #3 (BA3) and supplied to the display unit DP in the period T3. Writing is successively performed in units of macro blocks in writing into each bank, while the pixel data are successively read pixel by pixel in the scan order along the display system in reading from each bank. After completion of the period T3, the bank #4 (BA4) stores all pixel data of the B picture B2. Upon the completion of the period T3, display of the pixel data of the B picture B1 on the display unit DP is also completed.
In a period T4, encoded data of a P picture P6 are supplied and decoded, and the decoded pixel data are written in the bank #2 (BA2). In this period T4, the pixel data of the B picture B2 are successively read from the bank #4, supplied to the display unit DP and displayed thereon. Thereafter encoded data of a B picture B4, a B picture B5, a P picture P9 and a B picture B7 are sequentially supplied and decoded, and the decoded picture data are successively written in the banks #3 (BA3), #4 (BA4), #1 (BA1) and #3 (BA3) in periods T5, T6, T7 and T8 respectively. In parallel with such writing of the decoded picture data, the pixel data of the I picture I3, the B picture B4, the B picture B5 and the P picture P6 are read from the banks #1 (BA1), #3 (BA3), #4 (BA4) and #2 (BA2) respectively, supplied to the display unit DP and displayed thereon in the periods T5, T6, T7 and T8 respectively. The order of transmitting and decoding of the encoded pictures is made different from that of the displaying of the pictures, and therefore necessary future picture data are previously stored in the memory device MR when past and future picture data are employed in bidirectional prediction of B pictures, whereby the B pictures can be decoded.
As hereinabove described, the decoding/display unit having the display function employs the banks #3 and #4 for storing two frame pixel data, in order to convert the order of pixels (to convert the pixel data which are supplied in units of macro blocks into raster scan order). The storage capacity for two frames (pictures) is necessary for preventing pixel data which are not yet read for display from substitution by newly supplied decoded pixel data. For example, the I picture I3 is supplied in the period T1 and displayed in the period T5, and no pixel data can be stored in the bank #1 during these periods.
When the pictures are decoded and displayed at the same rate, it is possible to decode and display the pixel data within the same time by synchronizing starting of picture decoding with starting of picture display (output of pixel data from the memory device to the display unit). As for each B picture, display thereof is carried out in a period which is next to that for decoding As to the banks #3 (BA3) and #4 (BA4) storing the pixel data of the B pictures, one of these banks is subjected to writing of picture data while the other one is subjected to reading of picture data for display. In this memory device MR, therefore, substitution of data is completely prevented since memory areas for data writing are completely separated from those for data reading.
In the structure shown in FIG. 34, however, the memory device MR is required to have a storage capacity for storing pixel data of four frames. When a processed picture has resolution of the NTSC system (720 pixels by 480 lines), for example, pixel data of one frame are 720.times.480.times.8.times.1.5=3.96 Mbits, i.e., about 4 Mbits, assuming that one pixel data is formed by 8 bits. The coefficient 1.5 is employed in the calculation of the storage capacity, in consideration of such a state that the pixel number of two kinds of chrominance signals is half the pixel number of a luminance signal. In the conventional decoding/display unit, therefore, the storage capacity of the memory device is increased in order to convert the arrangement order of the display pictures, disadvantageously leading to increase of the device scale.